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ABSTRACT 

This report contains two papers that resulted from 
the Multilevel Evaluation Systems (MES) Project. The first paper is 
"Political and Practical Issues in Improving School Boards' Use of 
Evaluation Data" by Joan Herman* The second paper is "Principals: 
Their Use of Formal and Informal Data" by Shari Golan and Jeanne 
Dreyfus* The MES Project consisted of interview studies involving 
school board members, superintendents, and school principals. The 
study was designed to determine the sources of information and 
St idards used by these groups to judge the quality of their schools 
their interpretation and use of formal tests, and preferences for 
report content and format. The research was the result of a 
collaborative project involving the University of California (Los 
Angeles) Center for Research on Evaluation, Stanr^.ards, and Student 
Testing and participating district Research and Evaluation directors 
who conducted interviews with decision makers. Th {ES Project's 
multidisciplinary literature review and examinatioii of district 
reporting practices were the bases for hypotheses about how 
principals use data in the real world and the variables that are 
likely to influence such use. The two papers presented here provide 
the results of interviews with school board members and with 
elementary and secondary school principals. Numerous data tables and 
graphs are included. (TJH) 
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Introduction 



This report includes two papers which resulted from the 
Multilevel Evaluation System Project's interview studies. 

The first paper, Political and Practical issues in 
Improving School Boards' Use of Evaluation Data, presents the 
results of interviews with school board members. 

The second paper, Principals : Their Use of Formal and 
Informal Data, summarizes the results of interviews with 
elementary and secondary school principals. 

Both papers were presented at the Annual Meeting of the 
California Educational Research Association. They currently 
are being reivsed for wider dissemination. 
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Political and Practical Issues in Improving School Boards' 

Use of Evaluation Data 

Joan Herman 
UCLA Center for Research on • 
Evaluation^ Standards & Student Testing 

CRESST 



How do we make data useful for decisionmaking? For 
eons^ it seems^ many of us as evaluators have struggled with 
this question. We as data producers and data lovers see 
great potential for our efforts in improving educational 
policymaking and facilitating school improvement. 
Legislatures too, particularly in California, seem to agree 
on the power of evaluative information as they mandate school 
report cards, performance reports, and a full complement of 
tests. In the face of such optimism about the power of data, 
we see repeated evidence that the actual impact of our work 
is quite modest (Alkin, et al., 1979; Patton, 1986; Herman, 
1987) . 

Clearly, simply proclaiming the value of our work is an 
insufficient strategy for assuring that it reaches that 
potential. In an attempt to design more effective 
strategies, we at CRESST had a minor insight: let's stop 
talking to ourselves about the value of our work, let's stop 
talking about hypothetical impact if people like us were in 
planning and decisionmaking roles in schools, and let's start 
talking and better understcinding those who actually occupy 
those crucial roles in schools. We need to get inside the 
heads of real school decisionmakers and better understand 
where our data fits into their worlds. We need to design 
strategies that meet their needs, not our misconceptions 
about their needs. 

With these perspectives in mind, we embarked on 
interview studies of school board members, superintendents, 
and school principals. Our questions were relative simple: 
what sources of information do these groups really ^ to 
judge the quality of their schools, what standards ) they 
use to arrive av. their judgments; how do they make sense of 
and use the formal test data they receive, and what are their 
preferences for report content and format. In the process of 
answering these questions, we also developed interest in 
their basic affect toward and belief in testing. 

The research was the result of a unique collaborative 
project between the UCLA Center for Research on Evaluation, 
Standards, and Student Testing (CRESST) and participating 
district R&E directors who conducted interviews with 
decisionmakers within their local districts. It was a 
collaboration which had direct benefits for everyone 



involved. In the process of doing interviews, our colleagues 
gained visibility within their districts and the opportunity 
to better understand the needs of those they are supposed to 
he serving. And we gained key informants who are highly 
knowledgeable about local context and a very cost-effective 
c?iata bare about information use in a wide range of 
institutional settings. The na'.-ure of these settings and the 
specific methodology used in the project is described in the 
sections below, followed by a summary of results and their 
implications . 

Methodology 



The MLES project •s multidisciplinary literature review 
and examination of district reporting practices were the 
basis for hypotheses about how principals use data in the 
real world and the variables which are likely to influence 
such use. These were the basis for the sample design and the 
interview protocol. 

The Interview Protocol 

The interview protocol was developed to examine 
questions and variables of interest and revised based on 
field test results. The final version had three parts: 

Fre-interview. The pre-interview consisted of an 
introduction^ a statement of purpose, a description of the 
general content of the interview, an assurance of anonymity, 
permission to tape the interview, and solicitation of 
clarifying questions. 

Interview. The second part of the protocol contained 
the interview questions. The questions concerned the 
sources, criteria and application of information principals 
use to assess the quality of education in their schools. 
Other questions were about principals* interest in sub-groups 
test data, preferences for particular formats for test 
reports and additionally desired information. The final 
question asked principals how much they felt test data 
reflects what is important in schooling. Copies of the exact 
questions used are located in Appendix A. 

Post-interview. The final section of the protocol 
consisted of questions for the interviewer (which were 
completed as soon as possible after the interview) . This 
sheet had five comment areas — key themes, areas of concern, 
areas of confusion, personal reflections and suggestions for 
future interviews (see Appendix B) • 



Tha Subject Population 



Twelve districts were selected for participation in the 
study to represent a range of socio-economic status levels^ 
diversity with regard to ethnic composition, and a range of 
district sizes • Within each district, three principals and 
three board members were to be randomly selected for 
interviews. This paper focuses on the subsample of 27 board 
meribers who participated in the study. 

Interview Procedures 

Staff from the U.C.L.A. Center for Research on 
Evaluation, Standards, and Student Testing (CRESST) and 
directors of district research and evaluation offices were 
recruited and trained in study procedures. Training included 
discussion and practice in the use of the interview protocol 
as well as directions for randomly selecting respondents, and 
taping, summarizing and returning completed interviews. 
Interviewers then used the protocol to obtain data. 

Each interviewee was first contacted by phone and asked 
to be involved in the study. Most interviews took 
approximately a half hour. All the interviews were tape 
recorded. The interviews were all conducted between May and 
September, 1989. 

Interviewers summarized the results of their interview 
as soon as possible after each interview. Both the 
interviews summaries and tapes were turned over to the MLES 
staff. The MLES staff then coded and analyzed the results • 

Analysis of thG Data 

Tape interviews were summarized and a code book was 
assembled and tested y.-r four coders. Formal coding did not 
begin until there was one-hundred percent agreement on eight 
test interviews . 

For the study reported in this paper simple frequencies 
were tabulated. Small sample sizes precluded analyses based 
on contextual variables and likewise limit the 
generalizability of results. 



Rasults 

Who were the School Board Respondents? 

The respondents. Interviews were completed with 27 
school board members from a total of ten school districts. 
The obtained sample was fairly evenly distributed between 



those who were new to their school board roles (less than two 
years on their boards); those who and an intermediate level 
of experiences; and those whom we considered veteran school 
board members (six or more years on the board. (See Table 1) 

Their school district context. Based on state 
definitions, two of the ten districts represented are 
classified as large city school systems, two as small school 
districts, and the remaining eight as mid-sized districts. 

While the communities se rv6ci by th6 districts ar6 varied 
in SES levels^ the majority are characterized on average as 
mid-level SES^ with about one-quarter low SES and a single 
district serving a high SES community. The communities 
served are ethnically diverse. 

The most recent results of the California Assessment 
Program give a sense of the achievement context in which our 
respondents operate. More than 40 percent serve in districts 
where students score below the state average in reading and 
in math. 

In summary^ although the respondents and their districts 
were mostly from Southern California, we believe they 
generally represent the typical district in the state. It 
should be noted, however, that clearly absent from our study, 
are small rural districts. 

What Key Information Sources Are Used to Assess School 
Quality? 

Table 2 and Graph 1 show the sources of information 
school board members most frequently mentioned in evaluating 
the quality of their schools. Note that although almost all 
respondents mentioned testing as an important source, only 
about half were familiar enough with testing to be able to 
name a specific kinds of test (e.g., CAP, state assessment, 
etc . ) . 

The results in Table 2 and Graph 1 show a clear 
preference by school board members for qualitative, informal 
sources of information, often gained interpersonally . 
Consistent with both their often political roles and their 
positions as community liaisons, they rely heavily on 
feedback from parents, the community and the media. Such 
feedback apparently often comes on a "catch as catch-can" 
basis, e.g.f conversations with checkout clerks, interactions 
during social gatherings, etc. For example, as one veteran 
board member put it, "I get a lot of information in the 
grocery store, on the street, from people who work in the 
district who would talk to me off the record and away from 
their job site.*' 



School board members also highly value their own 
observations of classrooms and schools. Mentioned by almost 
three quarters of our respondents/ this source aJ so was the 
most likely of any source mentionea to considered critically 
important and heavily weighted in evaluations of school 
quality: \ 

The highest priority is going to a school — and checking 
the environment — the faculty and students and see how 
they are doing in the classroom. 

In some ways what I've found \o be most valu^^ble is the 
one-on-ones that I have as a board member out in the 
schools .. .primarily with students^ with teachers, with 
administrators and observations ... It ' s a real valuable 
source of information because so often, the information 
doesn't filter all the way to the board without that 
link . 

You see the kinds of effort the teachers are putting 
into their classrooms and I think that says a lot. If 
the classrooms are vibrant ^ and a lot of activity on the 
board — I think that's a good indicator that things are 
happening . 

Many school board members, in addition, appear to rely 
highly on the opinions of professionals in their districts. 
Almost three-quarters depend on the superintendent or other 
district administrators for perspectives on school quality. 
As mentioned later in this paper, many appear highly 
dependent on the school professionals to summarize and 
interpret data for them. One board member noted, "As far cxs 
numbers, I couldn't tell you whether being in the 60th 
percentile is o.k. or not. I have to rely on seasonec' 
administrators, including principals.'" In contrast, a small, 
vocal minority is highly skeptical of inforn.ation and 
opinions offered by their school district staffs. 

Drop-out rates, student grades, and results of special 
competitions and awards are valued by a sizeable minority of 
respondents. Relatively few pay attention to other more 
formal and quantitative informatJon sources, e.g., attendance 
rates, teacher turnover, results of coll'^ge entrance exams, 
formal surveys. It is of interest; to note the low frequency 
of formal survey data, even though surveys were routinely 
administered in several of the sampled districts. 

How Do School BoaTcd Membiars Judge Quality? 

Based on test 8Cor«^s. Almost all lespondents queried 
claim that they make determinations of school quality by 
looking for trends over time (see Table 3) , Scores going up 
indicate that their schools are doing well, while scores 
going down indicate that rhere are problems. Said one Board 



member, "If we've gone up, hopefully we've done the right 
thing. If we've gone down... lots of things could account for 
this." 



Respondents generally seemed to have difficulty, 
however, in articulating how large a change actually 
indicated progress in either direction. Even those who were 
able to articulate specific criterion, however, did so only 
tentatively, as this veteran's comment indicates: "if we 
made anywhere from between a 10 to 20 percent jump each year- 
~I think I'd be real happy .. .that ' s a guess." 

Also common, reported by about three-quarters of board 
member respondents, is the use of the normative standard of 
how other schools or districts are performing. Board members 
check to see whether their students' scores are above, below, 
or even with similar schools or districts. As one board 
member put it, "Of course you look at test scores and compare 
one school with other similar ones to see if you're living up 
to expectations." 

About half the respondents judge quality by looking to 
see whether students are scoring at or above the national or 
state average or whether they are "on grade level." This 
criterion may be less used than the above two because of 
where students in the sampled districts currently score. 
Recall that over 40% of respondents serve in districts which 
score below the state average on CAP. They perhaps achieve a 
sense of progress and success — and/or can convey such a 
vision publicly — by looking for improvement and relative 
accomplishment . 

It is of interest to note th'it few respondents, less 
than 20%, evaluate the quality of their schools in relation 
to specific expectations or goals, e.g., students scoring at 
specific levels; students mastering specific goals. 
Criterion-referenced interpretations or performance related 
goals are generally not in board members' vernacular. 

Based on personal observations of schools and 
classrooms. In making juU'^ *\ents about the quality of their 
schools, board members were about equally divided among those 
who did and did not claim to be assessing relatively specific 
aspects of the school setting. Consistent with the 
interpersonal orientation mentioned earlier, the most 
commonly observed aspect was school climate, as evidenced by 
the quality of interactions among and between students and 
teachers. Also commonly mentioned were teacher morale, 
teaching methods used in classrooms, and perceived student 
involvement in classwork (e.g., interest, attention). About 
one-fifth of the responded also paid some attention in their 
observations to classroom management and control issues. The 




6 



1 



flavor and variety of criteria used are evident in these 
conunents : 

I expect to see cooperation between teachers, I expect 
to see classrooms where students are attentive and are 
learning, I don't expect to hear a lot of non-school 
related noise — the playgrounds are well supervised, the 
lunch shelters are well organized and the kids are under 
control . 

Relatively infrequent were those who claimed to pay 
attention to physical characteristics — the appearance of 
teachers, classrooms, or school grounds; the presences of 
specific instructional resources. Also rare were those who 
paid attention to the nature or quality of student work 
products or instructional content, (See Table 4 and Graph 2) 

How Do School Board Menbars Make Sense of the 
Evaluation Reports They Receive? 

While a majority of school board members report 
scrutinizing the reports they receive, an even larger number 
report seeking help from others to help them understand and 
interpret the findings. One board member was very frank on 
this issue: "In a nutshell, I rely a lot on management to 
help me understand. After all as a trustee I don't know your 
job or other testers' job out there so I rely on that kind of 
information and support." 

About a half are looking for aberrations as they peruse 
findings, e.g., data that doesn't conform to general 
pattern?, findings that somehow jump out from the page. Or 
as one put it, "I look for something that is really out of 
whack . " 

About a third are looking to answer specific, although 
largely idiosyncratic, questions, e.g., how does school x in 
my region compare to school y? how many students scored 
above grade level? Has the proportion gone up? how did 
students perform in higher level skills. (See Table 5) 

In contrast to principals' use of results, less than a 
third are specifically loo/ing for areas of strength or 
weaknesses and only a couple mentioned that results were 
a rectly linked to calls for programmatic changes. Board 
members also were relatively less interested in subgroup 
performance. While almost two-thirds were interested in 
breakdowns of results by ethnicity, only about one third 
volunteered interest in specific results of limited English 
proficient students, and only about ten p . rcent expressed 
interest in gender differences or differences by program 
category. (See Table 6) 



What Are School Board Hembers Prefearences for 
Reporting? 

Content of Reports. The great majority of school 
board members had little to add when asked what additional 
information would be helpful to them. However, a significant 
minority, 20 to 25%, expressed interest in additional 
information, including quantitative indicators such as 
attendance rates and drop-out rates; and better information 
about school climate. (See Table 7) 

A slightly larger proportion also were interested in 
additional analyses that might contribute to their 
decisionmaking. Over a third wanted analyses that would help 
them compare the effects of different instructional programs, 
and about a quarter each wanted analyses that would let them 
see relationships between performance and student 
demographics and answer specific policy or practical 
questions . 

Format of Reports. Results regarding format indicate 
the difficulty of pleasing everyone. While half the 
respondents preferred graphical displays, about a third each 
listed tables or narrative as favored modes of communication. 
(See Table 8 and Graph 3) 

Whatever the format, however, brevity appears to be the 
valued characteristic. Among reactions were. 

Well, the first thing that pops into my head is that 
they are usually boring. I read the executive summary 
and that makes sense to me, and I suppose that it's just 
the nature of the beast that a lot of numbers are 
boring . 



A small, concise report — four or five pages at the most. 

Forty percent volunteered their desire for brief executive 
summaries of findings. However, because only ten percent 
wanted less technical information, it appears that while 
board members desire the findings digested for them, many 
also want to credibility of the full technical data. 

Implications and Conclusions 

The results of the interviews carry both good news and 
bad news for those concerned with the use of information and 
its effects on educational policy and program improvement. 

Good news. First, the findings help to allay some of 
the concerns of those who have reservations about the value 
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of standardized testing and who worry about over-reliance and 
misuse of test results • The results clearly indicate that 
for most school board members such worries are misplaced • 
Most school board members cli. im to be aware of the 
imperfections of tests and judge the quality of their schools 
based on a broad array of information sources • One board 
member put it this way: "I know our youngsters get a lot 
more out of school than, than just test scores •••so we can't 
go whole-hog on just test scores • I know that. There's 
other good stuff out there." 

In shorty board members' judgments appear to reflect a 
balance of both qualitative and quantitative data, much of 
which is experientially based. Consis"- rut with research in 
knowledge utilization (Lindblom and CohsSn, 1979), school 
board members seem to rely on informal working knowledge ever 
formal information. While most do not appear to over-rely on 
test data, some do seem to accept unquestioningly any test 
results . 

A second potential source of good news concerns the 
working relationships between the board members in the study 
and the education professionals in their respective 
districts. Board members generally rely on district 
administrators — the superintendent, the research and 
evaluation director, school principals — for their information 
about the quality of schools and trust these individuals to 
interpret available data. A vocal minority, however, 
expresses great skepticism and distrust of their 
administrators' opinions interpretations. Exemplifying this 
view was the comment, "I think that numbers can be put 
together in such a way that people are going to prove a 
prejudice or an idea that they had." Members of this second 
group do not believe anything unless they see and process it 
for themselves. 

A third piece of good news which emerges from the 
findings is that relatively simple changes in reporting 
strategies could increase the utility of information for 
board members. For those districts or states who do not 
already do so, the addition of an executive summary would 
make evaluation findings easier for board members to digest. 
"Keep it short" and "keep it simple" are too important 
dictums. Further, organizing nuch summaries (and the full 
reports) around the big questions which board members seek to 
answer would also be beneficial. The questions include: How 
are we progressing over time? How do our results compare 
with similar districts? How do our results compare with the 
national norm group? What are the implications of reported 
results for policy (areas of relative strength and weakness, 
differential performance by group or program) . 

Differences in Bocrd members report format preferences 
preclude easy prescription^^ . Backing executive summaries 



with technical data provided in all formats (graphics^ 
tabular^ and full narrative) is one possibility lor pleasing 
everyone^ although a weighty one. Combining an executive 
summary with simple^ visually clear graphical displays would 
probably please a majority^ assuming that full technical data 
were provided as an appendix. The tension among differing 
preferences^ however^ may be a relatively short term problem 
as desktop technology becomes even more accessible for all. 
For the future^ technology-based reports could enable 
everyone to customize reporting to their own preferences for 
both organization and format without having to wade through 
endless pages to find an appropriate match. 

Bad news. A first note of bad news relates to informed 
consumerism. While most board members claim not to 
overemphasize test results^ their knowledge and expectations 
for testing appear less than adequate. Only half of the 
respondents seemed to be aware that different kinds of tests 
were given in their district — at a minimum in these 
districts^ the state assessment program and a norm-referenced 
commercial test. Most appeared to talk in the vernacular of 
norm-referenced tests^ interested in whether students scored 
at or above grade level or at or above the national average^ 
but almost none seemed to recognize that their goals were 
unrealistic given the nature of such tests. "If we're at the 
50th percentile^ being average isn't acceptable to me." 
Measurement experts know that all students cannot score above 
the national average^ that all students cannot show progress 
relative to a national a\'erage^ that scoring "at grade level" 
does not have a clear task or skill referent; board members 
do not. If Board members are to seriously use test 
information^ they need to be helped to become more critical 
consumers . 

Some believe that education is best guided by a vision 
of what students should be able to do and accomplish; board 
members generally do not articulate such a vision or 
recognize the weaknesses of available norhv- referenced test 
information for evaluating progress toward such a vision. 
Further^ board members' qualitative bases of information are 
largely idiosyncratic, built on personal views of what 
constitutes a good educational program, a good school, a good 
climate, and drawn from relatively happenstance interactions 
and observations. While such perceptions certainly have face 
validity, the reliability of their samples are moot. Chance 
encounters at K-Mart check-out counters may or may not 
accurately represent the full reality. Further, the 
construct validity of board members judgments, particularly 
those derived from school and classroom observations, could 
be strengthen. Judgments could benefit from firmer grounding 
in research-based conceptions of educational quality and in 
research-based principles of effective schooling and quality 
instruction. While clearly it is unrealistic to expect 
Board members to be current on the research, it is possible 



10 



to envision training opportunities where Board members might 
refine what they look at/for as they observe and evaluate 
classrooms and schools. 

Attention to student performance is a prime candidate 
for emphasis should such opportunities occur. What are 
students working on? What is the nature of the tasks they 
are able to accomplish, the products they result in? Are 
these tasks and products meaningful, motivating, likely to 
require higher level thinking, likely to result in long-term 
learning? Board members as a group give little attention to 
such issues. Articulating the important elements of a 
productive school climate, given the attention board members 
pay to the climate issue, appears another priority. 

Is it reasonable to expect school board members to be 
more informed data users? Is it possible to help them become 
such users? The answer to both these questions is "probably 
yes" But in the end, the worst news which emerges from the 
experience of interviewing is an answer to the question "Will 
such use make a difference for educational quality?" In 
talking with Board members it seems clear that for most, the 
impact is likely to be marginal. Test scores and other data 
make a big initial splash when they appear in the newspaper 
or are bought to the Board. The splash may initiate 
consideration of the quality of education and how to improve 
it, but the splash is generally short-lived as a board moves 
on to the myriad of other matters which come before it . In 
fact, discussions of or agenda items related to educational 
quality are few and far between; board agendas appear swamped 
by the reality of approving consultant contracts for school 
workshops, for renovations, for new facilities and the like; 
and by the reality of maintaining status and position. In 
many of the schools and districts we visited, the problems of 
surviving, of maintaining the bureaucracy, of maintaining 
one's power, appeared to have displaced the primary goals of 
education . 

The use of data to improve schools assumes a largely 
rational, educational goals-based system. The technical 
problems of providing reasonable, useable data are tractable, 
and the results of our interviews provide some feasible 
directions for their solutions. Assuring their impact on 
educational quality, however, will require continuing 
attention to larger socio-political issues. 
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Table 


1: 


Background and Demographics 
School Board Members 
n = 27 




Experience 




% 


District 
Size 


% 


District 
SES 


% 


Beginning 


18 


.5 


Small 


11.1 


Low 


25.9 


Intermediate 


22 


.2 


Medium 


74.1 


Mid-Level 


66.7 


Veteran 


33 


.3 


Large 


14.8 


High 


7.4 



Table 2: Sources of Information Used by 

School Board Members 
n = 27 


Sources 


% 


Informal Parent Input 
Sch/Class Observation 


79.9 


70.3 


oupennb/uuner Adm inpur 


69 . 2 


Informal stu Input 


55.5 


Community Input 


55.5 


CAP 


51.8 


Informal Teach Input 


48.1 


norm jKeierence lesus 


44 . 4 


Newspapers 


42.3 


Principal Input 


38.5 


Grades 


33.3 


Dropout Rate 


33.3 


Awards 


25.9 


Discipline 


22.2 


College Prep Exams 


22.2 


Proficiency Tests 


18.5 


Teacher Turnover 


11.5 


Success - Next Level 


11.1 


Mobility Rate 


11.1 


Entries - Higher Level 


11.1 


Attendance Rate 


11.1 


Counselor Input 


7.7 


Formal Parent Input 


7.4 
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Table 3: How School Board 
Evaluate Test 
n = 27 


Members 
Scores 


% 


Trends Over Time 


88 .5 


Compare Similar School 


73.1 


Grade Equivalency 


50.8 


Compare Personal Goals 


18.5 


Indiv Student Scores 


3.7 
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Table 4: Criteria for Classroom Observations 

School Board Members 

Criteria % 

School Climate 48.1 

Student Involvement 37.0 

Teacher Morale 29.6 

Teaching Method 29.6 

Class Appearance 22.2 

Class Management 18.5 

Work Product 14.8 

School Appearance 14.8 

Class Resources ii.i 

Instructional Content 11. i 

Teacher Appearance 7.4 

Lesson Plans 3.7 
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GRAPH 2 

C'riteria for Classroom Observations 

School Board Members 



Sch Atmosphere^ 
Student invdv 
Teacher Morale ^ 
Teach iik; Method ^ 
Class Appear 
(".loss Managemt 
Wcrk Product 
Sc\\ A(jpe(J. p 
(.. lOss F^esoLif ces 
Insti Cofitent 
rr<j( ht-r Appear 
;oi I Plans 




1) 



40 



2 i-v 



Criteria 



CO 



80 



100 



Table 5: How School Board 
Make Sense of Test 
n ^ 27 


Members 
Scores 




Activity 


% 




Seek Help 


70 


.4 


Scruti lize Report 


55 


.6 


Look for Abber rat ions 


48 


.1 


Answer Questions 


33 


.3 


Skim Report 


33 


,3 


Look for Strengths/Weaknesses 


29 


. 6 


Communicate Results to Others 


22 


.2 


Summarizes Report 


18 


.5 


Seeks Instructional Reform 


7 


.4 
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Table 6: Interest in Sub-Group 
Differences 


School Board Members 
n = 27 






Ethnicltv 




Limited English 


36.0 


Social Economic Status 


29.6 


Stereotype Misuse 


14.8 


Gender 


11,1 


Achievement 


7.4 


Other Programs 


7.4 


General Interest 


7.4 



Table 7: Additional Information 

Wanted by Sch Board Members 
n = 27 


Information 


% 




Analysis of Test/Programs 


33 


.3 


Info on Individual Schools 


22 


.2 


Analysis of Test/Demographics 


22 


.2 


Analysis for Policy Questions 


22. 


.2 


descriptive info on Test 


18, 


.5 


Other Quantitative Indicators 


18. 


5 


School Climate Information 


14. 


8 


Prescriptive Information 


7. 


4 
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Table 8: Preferred Format 

School Board Members 
n = 27 



Format % 

Graphs 48.1 

Executive Summary 37.0 

Tables 33.3 

Narrative 30.8 

Better Keys 14.8 

Less Technical Info ll.i 
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Principals: Their Use of Formal and Informal Data 



In today's information age school principals are 
bombarded with enormous amounts of information about those 
whom they serve. Test scores^ attendance rates, discipline 
records, teacher, parent, and student input, and observations 
are just a few of the sources of information about students 
that principals receive. Given so many alternatives which 
ones do principals really use and for what purpose? Does 
one's preferences for different sources change according to 
how many years experience one has as a principal, whether one 
is an elementary, middle, or high school principal, or 
whether one's students are from low, middle, or high socio- 
economic standings? How can those of us interested in 
promoting the systematic use of data in schools help 
principals to use data more effectively? In this paper, we 
will begin to answer these questions and provide suggestions 
for improving the reporting of school data. 

The Multi-Level Evaluation System (MLES) project, funded 
by the Office of Educational Research and Improvement (OERI) 
and administered by the UCLA Center for Research on 
Evaluation, Standard, and Student Testing (CRESST) , has 
studied the use of school information for almost three years. 
The MLES project is investigating the feasibility of 
developing comprehensive information systems that will serve 
the planning and policy needs of school-based educators, 
district administrators, and school boards and intends to 
develop a set of design specifications for such systems. The 
first stage of the project was to compile a multi- 
disciplinary literature review that summarized the guiding 
principles for the design of school-based management 
information systems and the presentation of school data 
reports. Stage two featured a review of existing district 
reporting practices. This paper discusses some of the 
results of stage three, an interview study of 73 principals, 
school board members, and other district administrators about 
how they process and use information on school quality. 

We conducted a study that asked principals to define how 
they use school data and other information. We interviewed 
the principals using open-ended questions and carried out a 
content analysis of their responses. The procedures used in 
the interview and content analysis stages are described in 
the methodology section of this paper. Principals generated 
a long list of information sourcef; and some patterns of usage 
emerged. Some ways in which they misused information also 
appeared. The findings of the content analysis are discussed 
in the results section of this paper. The results of this 
study hold valuable lessons for those who create school data 
reports. The implication and conclusion section of this 
paper offers applications which can increase the utility and 
clarity of school data reports. 
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The MLES project's multidisciplinary literature review 
and examination of district reporting practices were the 
basis for hypotheses about how principals use data in the 
real world and the variables which are likely to influence 
such use. These were the basis for the sample design and the 
interview protocol • 

The Interview Protocol 

The interview protocol was developed to examine 
questions and variables of interest and then revised based on 
field test results. The final version had three parts: 

Pre-interview. The pre-interview consisted of an 
introduction^ a statement of purpose, a description of the 
general content of the interview, an assurance of anonymity, 
permission to tape the interview, and solicitation of 
clarifying questions. 

Interview. The second part of the protocol contained 
the interview questions. The questions CDncerned the 
sources, criteria and application of information principals 
use to assess the quality of education in their schools. 
Other questions were about principals' interest in sub-groups 
test data, preferences for particular formats for test 
reports and additionally desired information. The final 
question asked principals how much they felt test data 
reflects what is important in schooling. Copies of the exact 
questions used are located in Appendix A. 

Post-interview. The final section of the protocol 
consisted of questions for the interviewer (which were 
completed as soon as possible after the interview) . This 
sheet had five comment areas - key themes, areas of concern, 
areas of confusion, personal reflections and suggestions for 
future interviews (see Appendix B) . 

The Subject Population 

Twelve districts were selected for participation in the 
study to represent a range of socio-economic status levels, 
diversity with regard to ethnic composition, and a range of 
si^es. Within each district, three principals and three 
board members were to be randomly selected for interviews. 

This paper focuses on the subsample of 38 school 
principals from all 12 districts which were part of the 
study. Of the 12 school districts, two districts had less 
than 10 schools; eight had ten to 30 schools (classified as 
medium), and two had more than 30 schools (classified as 
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large) . The schools served in this obtained sample varied in 
ethnic composition and socio-economic level. The ethnic 
composition of most schools was either mixed-minority 
majority, mixed white majority or majority Latino schools. 
Other minorities represented were Afro-American and Asian. 

Three school levels ^^ere represented - eighteen 
elementary school principals^ eight middle school and eleven 
high school principals. Job experience data on twenty--six of 
the principals indicated that six were beginners with one to 
two years experience^ eight had three to six years 
experience^ and twelve were veterans with seven or more 
years • 

Intervisw Procadums 

Staff from the U.C.L.A. Center for Research on 
Evaluation^ Standards^ and Student Testing (CRESST) and 
directors of district research and evaluation offices were 
recruited and trained in study procedures. Training included 
discussion and practice in the use of the interview protocol 
as well as directions for randomly selecting respondents, and 
for taping, summarizing and returning completed interviews. 
Interviewers then used the protocol to obtain data* 

Each interviewee was first contacted by phone and asked 
to be involved in the study. Most interviews took 
approximately a half hour. All the interviews were tape 
recorded. The interviews were all conducted between May and 
September, 1989. 

Interviewers summarized the results of their interviews 
as soon as possible after each interview. Both the interview 
summaries and tapes were turned over to the MLES staff. The 
MLES staff then coded and analyzed the results. 

Analysis of the Data 

Tape interviews were summarized anc' a code book was 
assembled and tested by four coders. Formal coding did not 
begin until there was one-hundred percent agreement on eight 
test interviews. For the study reported in this paper 
frequencies were tabulated and analyzed for both overall 
principal data and on the basis of four specific contextual 
variables - the school's ethnicity and the school's SES, the 
principal's years of experience, and the school's performance 
on CAP reading scores. 

Limitations of the Data 

Like much of research conducted in natural settings^ 
this study had difficulties in controlling the presence of 
contextual variables of potential interest:. In addition, 



sample sizes limit the generalizability of findings^ 
particularly for those analyses which attempt to assess the 
effects of contextual variables^ e.g. principal's years of 
experience^ school SES^ and school CAP achievement. We also 
were not able to separate out the effects of school levels 
principal's experience^ school SES^ and school CAP 
achievement from each other • Due to these caveats^ the 
findings presented in our results section should be viewed as 
preliminary patterns and areas for future research. 



Results 

Wh^t Information Do Principals Use? 

Sources of information used. School principals who 
participated in our study use a total of twenty-four sources 
of information to judge the quality of education in their 
schools. The sources are both quantitative and qualitative. 
The median number of sources used by principals is eight. 

Our study found that the most frequently used source is 
school and classroom observation (nearly 87 percent utilize 
observations) . Many principals found classroom observations 
to be their most valuable source because observations provide 
immediate feedback as well as give the principals a wholistic 
and personal senso of how the school is functioning. For 
example^ one elementary school principal told us that 
classroom observations are his most valuable source because 
they provide "first hand information" that he can gather 
every day. Another principal said, a classroom observation 
"..•tells me more than anything else, so i visit at least 
five classrooms daily •" 

Other widely used sources included CAP scores, mentioned 
by seventy-four percent of the principals, and other norm- 
referenced tests mentioned by seventy-one percent. Although 
many principals use test scores when evaluating their 
schools, they feel these test scores receive more attention 
than they deserve because of the emphasis placed on them by 
state and district officials. Principals expressed their 
frustration about this over-emphasis of test scores. 
According to one principal, "Test scores are rated as 
important because of the State and District's emphasis on 
them." Another complained that, "With the over-emphasis of 
testing by bureaucrats, testing becomes an end, rather than a 
means to an end." Further, test scores seem to have their 
biggest impact as a source when they first are reported. A 
junior high school principal who mentioned test data as his 
first source of school information also said: 

Down the line, test data is not really at the top 
(important). It's only at the top when it's in the 
paper. When it's in the minds of people/ other than 
that, it's not a high priority. People forget about it. 
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CAP scores were also the most controversial source with 
sixteen percent of the principals saying that they have no 
value as a source of information. The principals complained 
about CAP'S instructions, self-report measures, and reporting 
of results in comparison bands (these bands are supposed to 
compare schools similar in socio-economic standing.) A high 
school principal who preferred CTBS to CAP told us, "The CAP 
test contains too many self-report items with bad and 
ambiguous instructions. Also, the CAP comparison bands are 
unfair because they do not take into account bussing." 

Informal teacher, parent and student input were 
mentioned as important sources by almost half of the 
principals and ranked fourth, fifth and sixth respectively 
among these mentioned. Student grades ranked a close 
seventh. Other quantitative sources such as teacher turnover 
rates, drop-out rates and mobility rates were among the least 
used. Thus while quantitative measures are viewed as 
important-ranking second and third in our study-qualitative 
indicators appear to receive more o/erall attention than 
quantitative indicators. (Detailed results of our analysis 
are summarized in Table 1 and Graph 1.) 

Effect of school level. The data yielded interesting 
contrasts depending upon grade level. There is a marked 
difference in what elementary and high school principals 
consider important sources of information. High school and 
middle school principals rely more heavily on informal 
teacher, student and parent input than do elementary school 
principals. Another difference is that elementary and middle 
school principals are more interested in CAP and other 
achievement tests while high school principals are more 
interested in college preparation exams such as the SAT. In 
addition, the use of classroom observation declines as the 
level of the school increases (see Table 2) . 

Effect of years of experience. Analyses by 
experience level based on a sub-sample indicate that those 
with six or more years of experience appear to use fewer 
sources of school information. Veterans of six or more years 
used a median of six sources while beginning principals with 
two or less years of experience used a median of nine 
sources. In addition, veterans appear to rely less on CAP 
and ncrm-referenced test data than do beginning principals. 
Furthermore, veteran principals are less likely to seek or 
use the advice of superintendents or other district 
administrators (such as research and evaluation directors) . 
Additional research would be necessary to validate these 
findings given the small sample size. (Refer to Table 3) . A 
difference in the emphasis placed on test data and district 
advice can be seen in these descriptions of the use of test 
data offered by a beginning and veteran elementary school 
principals : 



(Beginner) I rely on workshops given by the different 
project heads, such as tie CTBS or Director cf Federal 
Projects, to get an understanding of the district data 
and suggestions on how to present the data to my staff. 

(Veteran) I tend to be skeptical about test data... its 
just one piece of information about a child. 

E££ect of SES and CAP achiavament. The two 

contextual variables of socio-economic standing of th-^ 
student population and the school's performance on the 
reading section of the CAP exam were both related similarly 
to the use of sources. This was probably a function of a 
high correlation between SES anu success on the CAP exam. 
Four differences in the use of data emerged between 
principals from schools of different economic standings and 
levels of achievement on the CAP exam. Principals of schools 
with higher SES levels and which perform above the state 
average on CAP exams tend to be more interested in student 
success at the next grade or school level, attend more to 
informal parent and student input, and seek less guidance 
from district administrators such as research and evaluation 
directors than do principals of schools of lower SES students 
and those whose students perform below the state average on 
CAP. The lack of interest in administrator input among zhe 
high SES and achieving school principals may be due to uhe 
common tendency to not search out more information when 
feedback is good and supports prior expectations. (See Table 
4 and Table 5 . ) 

The differences in school evaluation approaches between 
low and high socio-economic elementary school principals are 
illustrated in the quotes below. The principal of a low- 
income based school that has been at the lowest percentile 
per grade level based on national norm, for years reported: 

To tell if my school is do-'ng j good job, I look first 
at test scores since they rre widely publicized. I want 
to see whether my student; are on grade level and 
whether there has been growth over the year. I feel 
testing is the most important objective source for 
determining student progress .. .My school has remained at 
about the same level test-wise for about five years. I 
consider this a plus in light of turnover in staff and 
the influx of new students. 

The principal of a high-income based school that consistently 
scores above the 90th percentile on standardized tests 
reported to us that to tell if her school is doing a aood 
job: 

...I observe classrooms. I want to know that the kids 
are learning and how they feel. I go into classrooms 



and look for interaction, active learning, and interest 
on the part of the students. I also receive parent 
feedback on the annual survey and from parents involved 
on the school site advisory committee. I also use test 
scores and find out how well students are doing in 
intermediate school. 

What Criteria Do Principals Use vman Making Their 
Assessments? 

Observing schools or classrooms. The most popular 
source of information named by principals was personal 
observations on both the school and classroom levels. Most 
principals, about two-thirds of the respondents, had specific 
areas of interest in mind when they made their observations. 
School or classroom climate was the most popular area of 
interest and was defined in terms of student-student and 
student-teacher interactions. When asked what he would find 
in a classroom with a positive climate, one principal named, 
"the teacher interacting positively with the children; a 
child who does something wrong is not put down — rather he is 
encouraged; and positive exchanges among the kids." 

Besides interactions among students and teachers, 
positive classroom climate was also defined in terms of the 
students' attitudes toward learning and the teachers' 
attitudes toward teaching. Students were seen as 
experiencing a positive classroom atmosphere "if students are 
happy, meaningfully involved, and like and understand what 
they are doing." Teachers were seen as creating a positive 
classroom atmosphere when they displayed "an attitude that 
education is important, that they love kids, that they want 
to impart this information." 

Other frequently mentioned areas of interest when 
observing a classroom were instructional content, teaching 
methods, student involvement in instruction, student work 
products, and teacher morale. (See Graph 2 for actual 
percentages of factors examined.) 

Analysing test data. Standardized tests such as CAP, 
CTBS, and others were the second and third most used sources 
of school information. One third of the principals 
interviewed reported needing help to interpret test results. 
Many more principals expressed that the time necessary to 
analyze test data thoroughly is limited and inconvenient. A 
junior high school principal explained: 

We're (principals) flooded with data. We are right now 
so caught up in the day-to-day management of our 
schools, we don't get to the point of analyzing 
data... The direction we're goinr means that this kind of 
data (test data) is going to mean less and less to us 



ERIC 



unless somebody comes along and pulls it out for us, 
condenses it, and summarizes it. 

For those principals who read and interpreted their own 
test results, their primary strategy was to look for trends 
of performance over several years (74 percent said they 
employed this strategy) . Half of the principals said they 
compared their school's performance to other similar schools. 
As one principal told us, "I look at average scores to see 
how my school compares with other schools, particularly 
schools that serve similar types of students. I also think 
that comparing student progress over the years — cohort 
tracking — is important." 

Also common when interpreting test results was comparing 
their school's performance to the national average or some 
other grade equivalency score. The desire to not fall below 
grade level on the national norm was strongly expressed. One 
principal told us, "We always strive and work toward being on 
grade level — always!" Besides the goal of being at or above 
grade level, 42 percent of the principals said they 
approached the data reports with predetermined goals for 
their schools, such as to be above the 75th percentile. (See 
Graph 3.) 

How Do Principals Apply Test Data? 

Using data. In addition to how principals make sense 
of test data, we were interested in how principals use and 
apply it. We found that the most common use of test data is 
to communicate it to teachers and parents (three-quarters of 
the principals reported doing so) . Also popular was using 
test data to identify areas of instructional strength and 
weakness. Slightly less frequent was follow-up to actual use 
of this data for instructional reforms. One principal told 
us that when she receives test results from the District: 

I return to my school and meet with the faculty all 
together and by grade level. In the grade level 
meetings we compare for each subject area teachers' 
scores with others, their strengths are recognized and 
areas in need of greater emphasis are identified. Ways 
to ameliorate a particular problem are discussed in 
detail. 

Over a third of the principals use the test data to 
answer specific programmatic questions. For instance, a 
principal may specifically check the movement of the third 
grade's CAP reading score because he or she is concerned 
about the introduction of a new basal reader in the second 
grade. Finally, almost a quarter of the principals mentioned 
they look for aberrations, unusual results, that may 
highlight potential problem areas. (See Graph 4.) 




Discrapancy handling. In order to determine to what 
extent principals rely upon test data and whether they use it 
uncritically^ principals were asked how they respond when 
faced with discrepancies between test data and other sources 
of data such as observations or input from teachers. Their 
answers suggest that principals are fairly sophisticated in 
their use of test data. Over one quarter of the principals 
said that they would consider multiple sources before 
determining the meaning of the discrepancy. When making 
sense of inconsistent information^ one principal told us how 
he involves his staff in considering these multiple sources 
of information. He said: 

The only solution to that (a discrepancy) is to analyze 
what the test is measuring and what is the population. 
We do this through a teacher committee and decide what 
information to pay attention to. 

We believe that this first approach to candling 
discrepancies^ considering multiple sources^ is the most 
appropriate and should therefore be further encouraged 
through in-service and pre-service trainings of school 
administrators . 

Another quarter of the principals said in the face of a 
discrepancy^ they would question the test. Other strategies 
mentioned by the principals were to question the match 
between the test's coverage and the classroom instructional 
coverage or to look to alternative explanations such as 
unusual weather or high rates of student mobility. This is 
what a principal who questioned the test as well as 
considered alternative explanations had to say: 

If there is a discrepancy between test scores and 
classroom observations^ I would look at changes in our 
student population^ especially in the number of LEP 
students. I would trust my observations over the test 
scores . 

Only one quarter of the principals said they w aid 
believe the test in the face of their own or others' 
observations. (See Graph 5.) For these principals^ the test 
results always had value^ and in this case reflected some 
defect in the instructional program. One such principal told 
us : 

We (the staff) have to go back and look and say^ What's 
going on here? Why do we have this? I don't think you 
can ignore this (the test results) . No matter what you 
think of the test; no matter how invalid you may think 
the test is; how prejudiced it is^ you are still 
competing against yourself on whether you go up or 
down. 
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It is interesting to note that the principals' reliance 
on and interpretation of test findings does not appear to 
vary with the school's economic standing, the years of 
experience at their job, the grade levels present at their 
school, or how well their school does on standardized tests. 
Therefore their handling of data discrepancies involving 
tests does not appear to be a defensive reaction. 

Are Principals Zntarested In Sub-Group Differences? 

Interest In sub-group dlf f (drences . Asked if they 
had any interest in sub-group differences, almost half the 
principals expressed an interest in performance differences 
among ethnic groups (primarily Latino, Afro-American, and 
White) . Another sub-group of interest was those students who 
are limited in English proficiency. Other bases for sub- 
groups of interest were socio-economic standing, gender, and 
levels of achievement . One tenth of the principals expressed 
the concern that sub-group information may lead to misuse and 
inappropriate stereotyping. One principal who expressed 
concern about reporting sub-group differences told us, "I 
don't believe we should pigeon-hole kids or pit this group 
against the other. Instead, we must look at lots of 
variables like home input and quality of teachers in that 
school . " 

What More Do Principals Want from School Reports? 

Additionally desired Information. Most of the 
principals seemed reluctant to ask for any more information 
than they already have or receive. Of those who were 
interested in additional information, the greatest interest 
was in analyses showing relationships between test results 
and specific instructional programs. These principals want 
to be able to tie test data back to their choice of 
textbooks, teaching methods, or other instructional reforms. 
Other desired information included: more data about 
individual students that could help with instructional 
diagnosis; more descriptive information about what the test 
is measuring; prescriptive information that tells the 
principal what instructional practices can be taken to 
rectify low scores in a particular area; information about 
other quantitative indicators such as dropout rates, mobility 
rates, and attendance rates; and analyses between test scores 
and student demographics (somewhat the same as sub-group 
differences) . Very few principals were interested in 
additional information about school climate although this was 
a great area of concern when discussing classroom 
observations. (See Graph 7) 

Preferred report formats. Principals reported their 
favorite format of presentation to be the graph, a finding 
which closely relates to their overwhelming interest in 
trends. This makes sense, because graphs are the clearest 
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way to display trends. Far fewer principals like narratives 
and tai^les. Other format preferences expressed were to 
include less technical information, to provide an executive 
summary with reports, and o improve the quality of keys for 
tables and graphs. (See Graph 8.) 



Implications and Concluslonft- 

This study set out to answer two basic questions: how do 
principals use school data and how can we help them to use 
that data more easily and accurately. This section of the 
paper addresses the latter question. It offers some 
recommendations for those people involved the creation of 
school information reports. The recommendations are broken 
down into two major categories: helping principals better 
analyze school data and helping principals better communicate 
with the public about the quality of their schools. 

Helping Principals To Analyze School Data 

Variety of sources. The first way in which one might 
help principals analyze school data is to make them aware of 
the full range of quantitative and qualitative information 
available to them. In our study principals identified 24 
sources of information which at least one principal, and 
usually more, found useful. Making principals aware of the 
full array may lead them to incorporate new sources of 
information into their own judgements about school quality as 
well as make them aware of sources that may be used to more 
concretely show what they already inherently believe about 
their schools. 

Evaluating the quality of test data. Most 
principals realize that test data represents only an 
estimation of how well their students are doing and that 
sometimes these estimations may be erroneous. However, a 
sizable minority of the principals interviewed in this study 
do not appear to be critical users of test data. Fully one- 
quarter accept without question the results of tests, even in 
the face of other evidence to the contrary. It would be 
beneficial for all principals to approach test results with 
skepticism and be fully familiar with their potential short- 
comings (such as a poor test, poor testing environment, 
mismatch of test and instructional goals) . Similarly, a 
number of principals appear to nee'^ assistance in more 
effectively using test results to improve their instructional 
prog IS, e.g., in identifying areas of instructional 
strengths and weakness and in designing and following up on 
program changes to address those weakness . 

Relating test data to instruction. Principals want 
help relating test data to instructional practices. When 
asked what additional information they wanted, nearly one- 



11 



third of the principals wanted analyses that could relate 
data to instructional reforms and/or other changes in 
curriculum and instruction. A similar number wanted explicit 
recommendations about what instructional changes should take 
place based on the test results. 

While both groups want help in better integrating 
testing with instruction, a distinction can be made between 
these two groups. Those principals who want to examine 
relationships between test results and instructional 
practices recognize that data can be used as a tool for 
decision-making. These principals appeared to have specific, 
albeit idiosyncratic, questions that they wanted the test 
data to help answer, e.g., has the new science textbook 
improved achievement? Have changes in grouping practices 
affected the performance of higher achieving students. This 
group of principals seems comfortable with using data and 
might benefit from automated, analytic tools to help them 
answer their questions. 

The second group of principals, however, appear to be 
looking for simple answers to complex professional problems. 
They want prescriptions — "just tell me what to do" appears to 
be their sentiment. This second group of principals might 
benefit from learning how to approach test data with specific 
questions that could help them in analyzing specific areas of 
instruction. For both groups, however, it would be wise to 
reinforce the ideas that large instructional decisions should 
not be based on test data alone. 

Formats. The choice of formats used in a school report 
can influence the degree to which principals can easily 
analyze and understand data. In this study, principals 
clearly state a preference for test data presented in terms 
of trends, and further report a strong preference for seeing 
those trends in graphical formats. 

Keys presented with graphs apparently are a source of 
trouble. Keys that accompany graphs should be able to stand 
alone and be easily understood. In some cases a brief 
statement summarizing the trend's interpretation also would 
be helpful. Finally, executive summaries that present key 
trends, identify aberrations, areas of strengths and 
weakness, and compare school results to other similar schools 
would prove helpful in many cases. 

Helping Principals to Communicate School Data to the 
Public 

Reporting to the public. Principals overwhelmingly 
use and value school and classroom observations more than 
test data in assessing the quality of their schools. They 
balance the use of qualitative and quantitative data. Then 
why should they use test data as the number one, and often 
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only, indicator of school quality when reporting to the 
public? The problem is that many principals do not know how 
to report qualitative data to the public in a way that is 
credible, defensible, and clearly understood. There are also 
issues about the objectivity of such observations. 
Developing observational protocols that can be more easily 
summarized as well as quantified might be a solution. Such 
protocols could operationalize each of the areas of concern 
discussed in this paper, incorporating the research base on 
these areas, and perhaps result in a bank of observation 
items. A principal could then select those areas of most 
concern to him or herself and develop a protocol on that 
basis. With the help of a research and evaluation director, 
the results of such a protocol could be summarized and 
quantified for the public. The public — and parents — also 
could be made privy to important elements assessed by the 
protocol so that they could make their own observations. 

Besides making observational information more credible, 
principalc need professional yet easy ways in which to 
gather, summarize, and report student, parent, and teacher 
feedback, perhaps through routine questionnaires or random 
phone interviews. Current attempts at school report cards 
will test the feasibility and validity of such practices. 

Concluding Rem&rks 

In conclusion, this study is an attempt to understand 
the use of data by school principals. Although it was based 
on a small sample size, the study suggests patterns of 
information usage that can be supported, expanded, and/or 
changed through specific action. The implications of this 
study can be applied immediately to the construction of 
school reports and the training of school administrators. We 
hope that this paper also will spur future theoretical 
studies of school information use and prove to be of 
practical help out in the real world of educational 
evaluation. 
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Table 1: Sources of 


Information 


Used 




by Principals 






Sources % 


Discounted 


% Used 


Rank 


Sch/Class Observation 


3 


87 


1 


CAP 


16 


74 


2 


Norm Reference Tests 


8 


71 


3 


Informal Teacher Input 


3 


58 


4 


Informal Parent Inout 


8 


53 


5 


Informal student Input 


3 


47 


6 


Grades 


- 


47 


6 


Formal Teacher Input 


- 


34 


8 


Counselor Input 


3 


21 


9 


Discioline 




24 


10 


Formal Student Input 


mm 


21 


11 


Community Input 


3 


21 


11 


Superint/Other Adm Input 




21 


11 


Attendance Rate 


5 


19 


14 


Proficiency Tests 




19 


14 


Newspapers 


5 


18 


16 


Success - Next Level 




18 


16 


College Prep Exams 




18 


16 


Entries - Higher Level 




13 


19 


Formal Parent Input 




13 


19 


Mobility Rate 




11 


21 


Teacher Turnover 




8 


22 


Dropout Rate 




8 


22 


Awards 




8 


22 
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Table 2: Percentages of Principals Who Use Sources 

By School Level 



Sources Elementary Middle High 

n = 18 n = 8 n = 11 
% % % 



Sch/class Observation 
CAP 

Norm Reference Tests 
Informal Teacher input 
Informal Parent Input 

Informal Student input 
Grades 

Formal Teacher input 
Counselor input 
Discipline 

Formal Student input 
Community input 
Superint/Other Adm Input 
Attendance Rate 
Proficiency Tests 

Newspapers 

Success - Next Level 
College Prep Exams 
Entries - Higher Level 
Formal Parent input 

Mobility Rate 
Teacher Turnover 
Dropout Rate 
Awards 



94 88 63 

50 100 36 

8 63 55 

39 75 73 

50 50 64 

28 75 55 

44 50 55 

39 25 27 

11 50 27 

22 50 9 

17 13 36 

17 25 18 

17 25 18 

17 25 18 

17 13 27 

11 13 27 

11 13 36 

0 0 64 

0 0 46 

11 13 18 

17 13 0 

11 0 9 

6 0 18 

6 13 9 



Table 3: Percentages of Principals Who Use Sources 

By Experience at Job 



Sources Beginner Intermediate Veteran 

n=6 n=8 n=l2 

% % % 



Sch/Class Observation 
CAP 

Norm Reference Tests 
Informal Teacher Input 
Informal Parent Input 

Informal student Input 
Grades 

Formal Teacher Input 
Counselor Input 
Discipline 

Formal student input 
Community Input 
Superint/Other Adm Input 
Attendance Rate 
Proficiency Tests 

Newspapers 

Success - Next Level 
College prep Exams 
Entries - Higher Level 
Formal Parent Input 

Mobility Rate 
Teacher Turnover 
Dropout Rate 
Awards 



83 


75 


58 


88 


75 


58 


63 


88 


67 


67 


50 


50 


50 


63 


58 


67 


25 


58 


67 


38 


50 


0 


38 


25 


50 


25 


33 


33 


25 


42 


17 


13 


25 


50 


13 


17 


50 


38 


17 


17 


25 


25 


17 


25 


8 


33 


25 


8 


17 


25 


8 


33 


0 


16 


17 


13 


0 


0 


25 


8 


0 


13 


8 


17 


13 


0 


17 


13 


8 


17 


13 


8 
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Table 4: Percentages 


of Principals Who use 


Sources 


By School SES 






Sources 


Low 


Middle 


High 




n = 6 


n = 14 


n = 4 
II t 




% 


% 


% 


Sch/Class Observation 


100 


79 


50 


CAP 


50 


93 


75 


Norn Reference Tests 


83 


43 


50 


Informal Teacher input 


50 


93 


75 


Informal Parent Input 


40 


64 


100 


Informal Student Input 


50 


71 


100 


Grades 


17 


50 


25 


Formal Teacher Input 


0 


0 


0 


Counselor Input 


17 


50 


25 


Discipline 


33 


36 


25 


Formal Student Input 


33 


29 


25 


Community Input 


17 


23 




Superint/Other Adm Input 


17 


43 


0 


Attendance Rate 


17 


29 


0 


Proficiency Tests 


17 


14 


0 


Newspapers 


0 


21 




Success - Next Level 


0 


14 


50 


College Prep Exams 


0 


14 


25 


Entries - Higher Level 


0 


14 


25 


Formal Parent Input 


0 


29 


25 


Mobility Rate 


17 


7 


0 


Teacher Turnover 


0 


0 


0 


Dropout Rate 


0 


0 


25 


Awards 


0 


7 


0 
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Table 5: Percentages of Principals who Use sources 


By 


CAP Scores 






Relative to CAP Reading state Average 


Sources 


Below 


At 


Above 




n = 9 


n « 4 


n = 5 




% 


% 


% 


Sch/Class Observation 


100 


79 


50 


CAP 


78 


100 


80 


Norm Reference Tests 


78 


25 


60 


Informal Teacher Input 


50 


93 


75 


Informal Parent Input 


40 


64 


100 


Informal studenc Input 


50 


71 


100 


Grades 


17 


50 


25 


Formal Teacher Input 


0 


43 


0 


Counselor InDUt 


17 


50 




Discipline 


33 


36 


25 


Formal Student Input 


33 


29 


25 


Community Input 


17 


29 


0 


Superint/other Adm Input 


17 


43 


0 


Attendance Rate 


17 


29 


0 


Proficiency Tests 


33 


0 


0 


Newspapers 


0 


21 


0 


Success - Next Level 


0 


14 


50 


College Prep Exams 


17 


14 


0 


Entries - Higher Level 


0 


14 


25 


Formal Parent Input 


0 


29 


25 


Mobility Rate 


17 


7 


0 


Teacher Turnover 


0 


0 


0 


Dropout Rate 


0 


0 


25 


Awards 


0 


7 


0 
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